Wire up external API for attaching external subnets #9781
Merged
+579
−78
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
bnaecker
commented
Feb 3, 2026
bnaecker
commented
- Plumb the existing app layer code to the new attach / detach sagas
- Add a bunch of integration tests confirming the new API behavior
- Closes Control plane support for attached subnets. #9453
- Add APIs to the sled agent for attaching and detaching either a single subnet on an instance, or setting / clearing the entire set for an instance. - Add list of attached subnets in the instance-creation request body, and fill that in from Nexus with the (currently-empty) set of attached subnets for the target instnace. - Plumb attachment requests all the way through the sled-agent internals to the new APIs in OPTE. - Add mapping of attached subnets per-instance to the simulated sled agent for testing. - Fixes #9702
- Adds a subnet attach and detach saga in Nexus, modeled after the existing floating IP attachment sagas. - Update the common code for sagas to include passing or removing the attached subnets to Dendrite and / or OPTE. This is to pick up those changes during the existing instance sagas, e.g. instance update. - Fixes #9685
Collaborator
Author
|
Stacked on #9780 |
bnaecker
force-pushed
the
nexus-attached-subnet-bg-task
branch
from
2867c38 to
8e62a71
Compare
- Plumb the existing app layer code to the new attach / detach sagas - Add a bunch of integration tests confirming the new API behavior - Closes #9453
bnaecker
force-pushed
the
wire-up-external-subnet-attach-api
branch
from
fe3ec22 to
7f2f485
Compare
Collaborator
Author
|
Ok, I've taken this for a spin on I installed the TUF repo from 7f2f485 using I then used a combination of the CLI and console to:
I then created a subnet pool and member: And an external subnet out of that: You can actually already use the CLI to do this with It shows up in the list of attached subnets for the instance now: The background task for this appears to be running, correctly propagating the subnet to Dendrite and OPTE: Next I'm going to see if we can actually transit packets through this thing. |
mergeconflict
approved these changes
Feb 4, 2026
| const INSTANCE_NAME: &str = "test-instance"; | ||
|
|
||
| #[nexus_test] | ||
| async fn test_instance_external_subnet_list_empty( |
Contributor
There was a problem hiding this comment.
It's a little hard for me to tell, is this old test subsumed by the new cannot_detach_subnet_that_is_not_attached?
Collaborator
Author
There was a problem hiding this comment.
I think so. The test test_external_subnet_attach has a few calls to this list endpoint for individual instances, e.g., L321 and L341.
Collaborator
Author
|
Last testing note, just to confirm that this actually works end-to-end. I spun up a new debian instance on Then I SSHd into the machine, and listened for pings while pinging from my dev laptop (which is on the VPN). From the laptop: On the VM: |
- Add tests ensuring we can attach subnets to running and stopped instances both. - Add test ensuring we can't delete subnet while it's attached - Cleanup, use max attached subnet const
|
Tested with two k3s nodes (Debian 13 Trixie), pods successfully getting real routable IPs from attached external subnets, communicating cross-node with no NAT and no overlay. Step 1: BGPCreated an address lot for Step 2: Subnet Pool & External SubnetsAdded two /26 members to the subnet pool from the BGP-announced range:
Created and attached external subnets:
Step 3: Transit IPsSet transit IPs on each instance's NIC matching their attached subnet:
No custom VPC router routes were needed. Step 4: CNI Configuration (each node)Installed standard CNI plugins and Whereabouts IPAM. Bridge CNI config on each node with Node 1 ( {
"cniVersion": "0.3.1",
"name": "oxide-pod-network",
"plugins": [
{
"type": "bridge",
"bridge": "cni0",
"isGateway": true,
"ipMasq": false,
"ipam": {
"type": "whereabouts",
"range": "172.20.35.0/26",
"gateway": "172.20.35.1",
"exclude": ["172.20.35.1/32"],
"routes": [
{ "dst": "0.0.0.0/0", "gw": "172.20.35.1" }
]
}
}
]
}Node 2 mirrors with Step 5: Host RoutesEach node needs a route to the other node's pod subnet via OPTE: Node 1: ResultsCross-node pod-to-pod ping: tcpdump on pod 2 confirms no NAT or overlay: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.